Image processing apparatus, monitoring system, image processing method, and program

ABSTRACT

A state acquisition unit (2020) acquires a state of a monitoring target in a captured image captured by a camera (3040). A monitoring point acquisition unit (2040) acquires, from a monitoring point information storage unit (3020), a monitoring point corresponding to the state of the monitoring target acquired by the state acquisition unit (2020). The monitoring point indicates a position to be monitored in the captured image. A presentation unit (2060) presents the monitoring point on the captured image.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/328,666, filed Jan. 24, 2017, which is a National Stage Entry ofInternational Application No. PCT/JP2015/065726, filed Jun. 1, 2015,which claims priority from Japanese Patent Application No. 2014-152336,filed Jul. 25, 2014. The entire contents of the above-referencedapplications are expressly incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to an image processing technique.

BACKGROUND ART

A method of monitoring a facility or the like includes a method ofperforming monitoring with viewing an image obtained by capturing animage of the facility or the like using a monitoring camera(hereinafter, a monitoring image). A technique for facilitatingmonitoring using a monitoring camera has been developed. For example,Patent Document 1 discloses a monitoring system that presents a framearound each item to be checked (door or the like) on a monitoring imagedisplayed on a monitor.

RELATED DOCUMENT Patent Document

[Patent Document 1] Japanese Unexamined Patent Application PublicationNo. 2001-054099

SUMMARY OF THE INVENTION

A monitoring target that an observer has to monitor is not necessarilythe same and may vary depending on the situation even when the sameplace in captured on monitoring images. However, Patent Document 1 doesnot take into account the fact that a monitoring target to be monitoredmay vary depending on the situation.

The invention is contrived in view of the above-mentioned problem, andan object thereof is to provide a technique capable of appropriatelyascertaining a place to be monitored in a captured image.

There is provided a first image processing apparatus including: a stateacquisition unit acquiring a state of a monitoring target in a capturedimage captured by a camera; a monitoring point acquisition unitacquiring a monitoring point corresponding to the state of themonitoring target acquired by the state acquisition unit, the monitoringpoint indicating a position to be monitored in the captured image, andbeing acquired from a monitoring point information storage unit storingthe state of the monitoring target and the monitoring point inassociation with each other; and a presentation unit presenting themonitoring point acquired by the monitoring point acquisition unit on afirst captured image captured by the camera.

There is provided a first monitoring system including the first imageprocessing apparatus, the camera, and the monitoring point informationstorage unit of the invention.

There is provided a first image processing method performed by acomputer. The method including: acquiring a state of a monitoring targetin a captured image captured by a camera; acquiring a monitoring pointcorresponding to the state of the monitoring target acquired by the stepof acquiring the state of the monitoring target, the monitoring pointindicating a position to be monitored in the captured image, and beingacquired from a monitoring point information storage unit storing thestate of the monitoring target and the monitoring point in associationwith each other; and presenting the monitoring point acquired by thestep of acquiring the monitoring point on a first captured imagecaptured by the camera.

There is provided a first program that causes a computer to operate asthe first image processing apparatus of the invention by causing thecomputer to have functions of functional components included in thefirst image processing apparatus of the invention.

There is provided the second image processing apparatus including: astate acquisition unit acquiring a state of a monitoring target in acaptured image captured by a camera; a monitored position acquisitionunit acquiring a monitored position, which is a position monitored by anobserver viewing a display screen on which the captured image isdisplayed; and a monitoring point information generation unit handlingthe monitored position as a monitoring point, and generating monitoringpoint information by associating the monitoring point with the state ofthe monitoring target, the monitoring point indicating a position to bemonitored in the captured image.

There is provided a second monitoring system including the second imageprocessing apparatus and the camera of the invention.

There is provided a second image processing method performed by acomputer, the method including: acquiring a state of a monitoring targetin an image captured by a camera; acquiring a monitored position, whichis a position monitored by an observer viewing a display screen on whichthe captured image is displayed; and handling the monitored position asa monitoring point, and generating monitoring point information byassociating the monitoring point with the state of the monitoringtarget, the monitoring point indicating a position to be monitored inthe captured image.

There is provided a second program that causes a computer to operate asthe second image processing apparatus of the invention by causing thecomputer to have functions of functional components included in thesecond image processing apparatus of the invention.

According to the invention, it is possible to provide a techniquecapable of appropriately ascertaining a place to be monitored in acaptured image.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects described above, and other objects, features and advantagesare further made more apparent by suitable embodiments that will bedescribed below and the following accompanying drawings.

FIG. 1 is a block diagram illustrating an image processing apparatusaccording to a first exemplary embodiment.

FIG. 2 is a flow chart illustrating a flow of processing that the imageprocessing apparatus according to the first exemplary embodimentperforms.

FIG. 3 is a block diagram illustrating a hardware configuration of theimage processing apparatus.

FIGS. 4A and 4B are diagrams illustrating a method of presenting amonitoring point.

FIG. 5 is a diagram conceptually illustrating a process of generatingstate information for each presentation target image.

FIG. 6 is a diagram conceptually illustrating a process of presenting acommon monitoring point with respect to a plurality of presentationtarget images.

FIGS. 7A and 7B are diagrams illustrating the presentation of amonitoring point based on the degree of congestion.

FIGS. 8A and 8B are diagrams illustrating the presentation of amonitoring point based on the movement of a monitoring target.

FIGS. 9A and 9B are diagrams illustrating the presentation of amonitoring point based on the state of a queue.

FIGS. 10A and 10B are diagrams illustrating two captured images capturedby two different cameras

FIG. 11 is a block diagram illustrating an image processing apparatuswith its usage environment according to a second exemplary embodiment.

FIG. 12 is a diagram conceptually illustrating that a monitoring pointto be presented is determined on the basis of an observer's eye gazedirection.

FIG. 13 is a flow chart illustrating a flow of processing that the imageprocessing apparatus according to the second exemplary embodimentperforms.

FIG. 14 is a flow chart illustrating a flow of processing that an imageprocessing apparatus according to a third exemplary embodiment performs.

FIG. 15 is a block diagram illustrating an image processing apparatusaccording to a fourth exemplary embodiment.

FIG. 16 is a flow chart illustrating a flow of processing that the imageprocessing apparatus according to the fourth exemplary embodimentperforms.

DESCRIPTION OF EMBODIMENTS

Hereinafter, exemplary embodiments of the invention will be describedwith reference to the accompanying drawings. In all of the drawings,like reference numerals denote like components, and descriptions thereofwill not be repeated.

In addition, the term “observer” used below generally refers to a personwho views a captured image (which is described later) and performsmonitoring, and is not limited to a person given a position named“observer”.

First Exemplary Embodiment

FIG. 1 is a block diagram illustrating an image processing apparatus2000 according to a first exemplary embodiment. In FIG. 1, an arrowindicates a flow of information. Further, in FIG. 1, each blockindicates a function-based configuration instead of a hardware-basedconfiguration.

The image processing apparatus 2000 presents a position to be monitoredon an image obtained by capturing an image of a monitoring target with acamera 3040, in accordance with the state of a monitoring target such asa person (whether or not the monitoring target is moving, or the like).For this purpose, the image processing apparatus 2000 includes a stateacquisition unit 2020, a monitoring point acquisition unit 2040, and apresentation unit 2060. The state acquisition unit 2020 acquires thestate of a monitoring target in an image that the camera 3040 captures(hereinafter, a captured image). The camera 3040 is, for example, amonitoring camera or the like. For example, the captured image is aframe constituting a video that the camera 3040 captures. Note that,details of a monitoring target and the state of the monitoring targethandled by the image processing apparatus 2000 will be described later.

The monitoring point acquisition unit 2040 acquires a monitoring pointfrom a monitoring point information storage unit 3020 using the state ofthe monitoring target acquired by the state acquisition unit 2020. Themonitoring point may be information indicating a position to bemonitored in a captured image. Note that, the monitoring point may be apoint that does not have an area or may be a region having an area. Themonitoring point information storage unit 3020 stores the state of themonitoring target and the monitoring point in association with eachother. The monitoring point acquisition unit 2040 acquires from themonitoring point information storage unit 3020 a monitoring point beingassociated with the state of the monitoring target that the stateacquisition unit 2020 acquires. Note that, the number of monitoringpoints associated with a certain “state of a monitoring target” may beone or two or more.

The presentation unit 2060 presents a monitoring point, which themonitoring point acquisition unit 2040 acquires, on a capture imagecaptured by the camera 3040. Hereinafter, a captured image on which amonitoring point is presented will also be described as a presentationtarget image.

A monitoring system 3000 is a system including the image processingapparatus 2000 and the camera 3040. The monitoring system 3000 mayfurther include a display screen 3060. The display screen 3060 is ascreen displaying a captured image that the camera 3040 captures. Inaddition, the display screen 3060 may display a presentation targetimage on which an indication of a monitoring point is presented by thepresentation unit 2060. For example, the display screen 3060 isconfigured such that it performs switching between a mode in which acaptured image captured by the camera 3040 is displayed as is and a modein which an image processed by the image processing apparatus 2000 isdisplayed. As described later, a presentation target image processed bythe image processing apparatus 2000 is not limited to being output tothe display screen 3060.

<Flow of Processing>

FIG. 2 is a flow chart illustrating a flow of processing that the imageprocessing apparatus 2000 according to the first exemplary embodimentperforms. In step S102, the state acquisition unit 2020 acquires thestate of a monitoring target in a captured image. In step S104, themonitoring point acquisition unit 2040 acquires from the monitoringpoint information storage unit 3020 a monitoring point corresponding tothe state of the monitoring target that the state acquisition unit 2020has acquires. In step S106, the presentation unit 2060 presents themonitoring point that the monitoring point acquisition unit 2040acquires, on a presentation target image.

<Operational Advantages>

According to the image processing apparatus 2000 of the presentexemplary embodiment, an indication of a monitoring point correspondingto the state of a monitoring target is presented on a captured image.Thus, according to the image processing apparatus 2000, a place to bemonitored is appropriately presented in a monitoring image or the like,depending on the state of a monitoring target. Accordingly, an observeror the like monitoring a monitoring image or the like can appropriatelyascertain a place to be monitored in a monitoring image or the like.

Hereinafter, the image processing apparatus 2000 of the presentexemplary embodiment will be described in more detail.

<Example of Hardware Configuration>

Each functional component of the image processing apparatus 2000 may beconfigured of hardware constituent elements (for example, an hard-wiredelectronic circuit or the like) which achieves each functionalcomponent, or may be configured of a combination of hardware constituentelements and software constituent elements (for example, a combinationof an electronic circuit and a program for controlling the electroniccircuit, or the like).

FIG. 3 is a block diagram illustrating a hardware configuration of theimage processing apparatus 2000. The image processing apparatus 2000includes a bus 1020, a processor 1040, a memory 1060, a storage 1080,and an input-output interface 1100. The bus 1020 is a data transmissionchannel for allowing the processor 1040, the memory 1060, the storage1080, and the input-output interface 1100 to transmit and receive datato and from each other. Here, a method of connecting the processor 1040and the like to each other is not limited to a bus connection. Theprocessor 1040 is an arithmetic processing device such as, for example,a central processing unit (CPU) or a graphics processing unit (GPU). Thememory 1060 is a memory such as, for example, a random access memory(RAM) or a read only memory (ROM). The storage 1080 is a storage devicesuch as, for example, a hard disk, a solid state drive (SSD), or amemory card. In addition, the storage 1080 may be a memory such as a RAMor a ROM. The input-output interface 1100 is an input-output interfacefor allowing the image processing apparatus 2000 to transmit and receivedata to and from the monitoring point information storage unit 3020 oranother external apparatus. For example, the image processing apparatus2000 acquires information indicating the state of a monitoring targetand a monitoring point through the input-output interface 1100. Inaddition, for example, the image processing apparatus 2000 outputs apresentation target image presenting an indication of a monitoring pointthereon, through the input-output interface 1100.

The storage 1080 stores a program for implementing the function of theimage processing apparatus 2000. Specifically, the storage storesprogram modules for implementing the respective functions of the stateacquisition unit 2020, the monitoring point acquisition unit 2040, andthe presentation unit 2060. The processor 1040 realizes the functions ofthe state acquisition unit 2020, the monitoring point acquisition unit2040, and the presentation unit 2060 by executing the respective programmodules. Here, the processor 1040 may read out the modules on the memory1060 and then execute the modules at the time of executing the modules,or may execute the modules without reading out the modules on the memory1060.

The hardware configuration of the image processing apparatus 2000 is notlimited to the configuration illustrated in FIG. 3. For example, eachprogram module may be stored in the memory 1060. In this case, the imageprocessing apparatus 2000 may not include the storage 1080.

<Details of Monitoring Target>

There are various monitoring targets that the image processing apparatus2000 handles. For example, the image processing apparatus 2000 handlesobjects such as a person, a thing or a set of objects (crowd or thelike), as monitoring targets. Note that, an object indicating a thingmay include a place. In other words, the image processing apparatus 2000may handle a place (region) in a captured image as a monitoring target.

For example, when region included in a captured image is divided into aforeground region and a background region, an object is a foregroundregion. Here, the object is not limited to the above-mentionedforeground region. There are various well-known methods of extracting anobject such as a person or a thing from an image, and these well-knowntechniques can be appropriately used to extract an object that the imageprocessing apparatus 2000 handles. Here, such well-known techniques willnot be described.

A monitoring target may be either all of objects extracted from acaptured image or may be only a specific object. For example, the imageprocessing apparatus 2000 handles only a person or a set of people(crowd) as a monitoring target. In addition, the image processingapparatus 2000 may handles only a specific person or crowd as amonitoring target. The specific object is specified by, for example,information showing a monitoring target. The information showing amonitoring target is information showing a list indicating afeature-value of the specific object (for example, a blacklist) orinformation showing features of an object to be monitored such as“wearing a hat” or “wearing sunglasses”. Here, since a technique ofextracting an object having a specific feature from objects in an imageis a well-known technique, a detailed method will not be described.

<Method of Acquiring State of Monitoring Target>

The state acquisition unit 2020 acquires information indicating thestate of a monitoring target (hereinafter, state information). The stateinformation needs to be information capable of specifying to whichmonitoring target an indicated state is related. Here, the monitoringtarget is represented by, for example, a combination of “an ID of acaptured image, coordinates on the captured image, and the type of amonitoring target”. In this case, the state information includes “an IDof a captured image, coordinates on the captured image, the type of amonitoring target, and the state of the monitoring target”.

In addition, when there are a plurality of cameras 3040, it may bedistinguished which camera 3040 captures a monitoring target. In thiscase, state information further indicates the ID of the camera 3040, orthe like.

The state information may be generated by the image processing apparatus2000, or may be generated by an external apparatus different from theimage processing apparatus 2000. When the state information is generatedby the image processing apparatus 2000, the image processing apparatus2000 acquires a captured image that the camera 3040 captures. Itextracts a monitoring target from the captured image, and calculatesstate information indicating the state of the extracted monitoringtarget. A specific method of calculating state information will bedescribed later.

Note that, when the state information is generated by an externalapparatus, a process of acquiring the state information may be a processin which the state acquisition unit 2020 receives the state informationoutput by the external apparatus, or may be a process in which the stateacquisition unit 2020 reads out the state information from the externalapparatus.

<Details of Presentation Unit 2060>

<<How to Indicate Monitoring Point>>

As described above, the presentation unit 2060 presents a monitoringpoint on a captured image that the camera 3040 captures (presentationtarget image). Here, there are various methods for the presentation unit2060 to present a monitoring point. For example, the presentation unit2060 presents a monitoring point by presenting a frame line or the likearound the monitoring point. In addition, for example, the presentationunit 2060 presents a monitoring point by changing the color of themonitoring point or the color of an area around the monitoring point.Note that, a process of “changing the color of a monitoring point or thelike” includes various processes such as: a process of changing thebrightness, contrast, or the like of the color of the monitoring pointor the like; a process of blending another color with the original colorof the monitoring point or the like; a process of painting themonitoring point or the like with a specific color; and so on. Inaddition, the presentation unit 2060 may present a frame line or thelike around a monitoring point and change the internal color thereof, ormay blink the frame line or a region the color of which is changed.

FIGS. 4A and 4B are diagrams illustrating a method of presenting amonitoring point. In FIGS. 4A and 4B, a monitoring point indicates anentrance 20. In FIG. 4A, the monitoring point is presented by presentinga frame line 30 around the entrance 20. On the other hand, in FIG. 4B,the monitoring point is presented by painting the entrance 20 with aspecific color. Note that, for convenience in drawing, painting with aspecific color is represented by filling with a dot pattern.

Here, the phrase “presenting a monitoring point on a presentation targetimage” refers to, for example, a process of embedding the monitoringpoint in the presentation target image or overlaying the monitoringpoint on the presentation target image. In this case, the presentationunit 2060 may output the presentation target image having an indicationembedded therein to an output device such as the display screen 3060 orthe like, or may store the presentation target image in a storage deviceprovided inside or outside of the image processing apparatus 2000. Inthe latter case, the display screen 3060 or another device reads thepresentation target image stored in the storage device and outputs it tothe display screen 3060. Note that, the display screen 3060 is, forexample, a monitor installed in a room or the like of an observer, amonitor of a mobile terminal of a security guard who performs guardingin the scene, or the like.

In addition, the presentation unit 2060 may separately generate imagedata presenting an indication which indicates a monitoring point,without embedding the indication in a presentation target image. In thiscase, the indication is presented on the presentation target image bydisplaying the image data together with presentation target data.

In addition, the presentation unit 2060 may present a monitoring pointon a map by using map data of a facility in which the camera 3040 isinstalled. The map data is displayed on the display screen or themonitor of a security guard's mobile terminal or the like. The positionof a monitoring target on the map can be calculated on the basis ofvarious parameters of the camera 3040 (the installation position of thecamera 3040, the orientation of the camera 3040, and the like and theposition of the monitoring point (coordinates) on a captured image. Inthis case, the presentation unit 2060 acquires and uses map data of thefacility in which the camera 3040 is installed and various parametersrelated to the camera 3040. Note that, a relationship between thevarious parameters of the camera 3040 and the position of the camera onthe map is defined in advance by performing calibration or the like.

<Processing in a Case of Acquiring Plurality of Monitoring Points>

Suppose that a plurality of monitoring points have been acquired by themonitoring point acquisition unit 2040. In this case, the presentationunit 2060 may present all of the acquired monitoring points on apresentation target image or may present only some of the acquiredmonitoring points. When only some of the acquired monitoring points arepresented, for example, priority is given to a monitoring point storedin the monitoring point information storage unit 3020. The presentationunit 2060 preferentially presents a monitoring point having higherpriority among the plurality of acquired monitoring points. For example,an upper limit number of the number of monitoring points that can bepresented on one presentation target image is provided. When the numberof acquired monitoring points exceeds the upper limit number, thepresentation unit 2060 presents the monitoring points equal to or lessthan the upper limit number on the presentation target image indescending order of the priority. The upper limit number is stored inadvance in the storage unit provided inside or outside of the imageprocessing apparatus 2000.

<<Relationship Between Presentation Target Image and Image Used toGenerate State Information>>

A presentation target image on which a monitoring point corresponding tocertain state information is presented may be a captured image used togenerate the state information, or may be another captured images. Inthe former case, for example, the presentation unit 2060 presents amonitoring point corresponding to state information generated usingfirst to n-th captured images on the n-th captured image. In the lattercase, for example, the presentation unit 2060 presents a monitoringpoint corresponding to state information generated using the first ton-th captured images on an (n+1)-th captured image.

For example, state information is generated for each presentation targetimage. FIG. 5 is a diagram conceptually illustrating a process ofgenerating state information for each presentation target image. In FIG.5, the presentation unit 2060 presents a monitoring point correspondingto state information generated using first to n-th captured images on an(n+1)-th captured image. Similarly, the presentation unit 2060 presentsa monitoring point corresponding to state information generated usingthe second to (n+1)-th captured images on an (n+2)-th captured image,and presents a monitoring point corresponding to state informationgenerated using the third to (n+2)-th captured image on an (n+3)-thcaptured image.

In addition, for example, the presentation unit 2060 may use stateinformation generated using a plurality of certain captured images incommon with respect to a plurality of presentation target images. FIG. 6is a diagram conceptually illustrating a process of presenting a commonmonitoring point with respect to a plurality of presentation targetimages. In FIG. 6, the presentation unit 2060 presents a monitoringpoint corresponding to state information generated using first to n-thcaptured images on each of (n+1)-th to 2n-th captured images. Similarly,the presentation unit 2060 presents a monitoring point corresponding tostate information generated using (n+1)-th to 2n-th captured images oneach of (2n+1)-th to 3n-th captured images.

<Specific Example of Monitoring Point>

A monitoring point is determined in advance in association with thestate of a monitoring target. Hereinafter, a specific example showingcorrespondence between the state of a monitoring target and a monitoringpoint will be described.

<<Case where State of Monitoring Target is Degree of Congestion ofCrowd>>

For example, a place to be monitored may vary depending on the degree ofcongestion of a place in a captured image 10. For example, when thereare few people in a place, it is considered that an entrance or exit tothe place (door, an end of an image, or the like) should be monitoredsince a person who will enter the place in the future should beobserved. On the other hand, when the place is crowded, it is consideredthat a place into which people gather, a place through which peoplepass, and the like should be monitored. When there is a very narrowpath, the flow of a crowd tends to be congested there, and thus it isnecessary to monitor such a narrow path. Thus, the presentation unit2060 presents a monitoring point based on the degree of congestion of aplace in the captured image 10. In addition, even when the degree ofcongestion is the same, there is a place where risk increases suddenlywhen it becomes difficult for people in a crowd to watch step due tocongestion, such as a place having steps. Such a place may be presentedas a monitoring point.

FIGS. 7A and 7B are diagrams illustrating the presentation of amonitoring point based on the degree of congestion. In the case of FIG.7A, no person is shown in a place captured in a captured image 10, andthus there is no congestion. Thus, an entrance 20 is a monitoring point.The presentation unit 2060 therefore displays a frame line around theentrance 20.

On the other hand, in the case of FIG. 7B, a crowd 50 is shown in acaptured image 10, and there is congestion. Thus, a narrow path 40 is amonitoring point. The presentation unit 2060 therefore displays a frameline in the narrow path 40.

Note that, correspondence between the above-mentioned degree ofcongestion and a monitoring point is merely an example. Thus, “amonitoring point is not necessarily an entrance when there is an enoughspace”, and “a monitoring point is not necessarily a narrow path whenthere is congestion”. For example, it could be determined that “amonitoring point is an entrance when a floor is congested with peopleattempting to leave a facility since a flow of people could become stuckat the entrance”. A concrete correspondence between the degree ofcongestion and a monitoring point is determined in advance on the basisof the characteristics of the place, the characteristics of people usingthe place, and the like.

Monitoring point information generated on the basis of the determinationis stored in the monitoring point information storage unit 3020.

Note that, although the degree of congestion indicated in stateinformation is divided into two cases of “being congested” and “notbeing congested” in the above-described example, the degree ofcongestion indicated in a monitoring point may be divided into three ormore levels. In this case, the monitoring point information storage unit3020 stores a monitoring point in accordance with each of three or morelevels of degree of congestion.

In the above-described example, although the degree of congestionindicated in state information is the degree of congestion for theentire place in the captured image 10, state information may indicatethe degree of congestion for a partial region in the captured image 10.For example, state information indicates the degree of congestion foreach of the vicinity of the entrance and the vicinity of the path.Thereby, as the monitoring point information, a monitoring point can bedetermined corresponding to the degree of congestion of each of thepartial regions, such as “the case where an entrance is congested” or“the case where a path is congested”.

For example, the degree of congestion can be calculated on the basis ofthe number of objects in the captured image 10. Specifically, acongested state is a state where the number of objects in the capturedimage 10 is equal to or greater than a pre-determined number. As thepre-determined number, a common value may be used for each place, or adifferent value may be used for each place. Note that, since a method ofcalculating the number of objects in the captured image 10 iswell-known, the detailed description thereof will not be described here.

<<Case where State of Monitoring Target is Movement State of Crowd>>

For example, a place to be monitored may vary depending on the movementof a monitoring target in a captured image. For example, when two crowdsare captured in a captured image and it is possible that they eventuallyintersect each other after they move, it is considered that it ispreferable to monitor a position at which the crowds are expected tointersect each other in the future. Thus, the presentation unit 2060presents a monitoring point based on the movement of a monitoring targetin a captured image.

In addition, even when the crowds do not intersect each other becausethe flows of the crowds have the same direction, a dangerous situationmay occur when the rear crowd catches up to the front crowd. Forexample, when the crowds flow in at interval, the slow flow of the frontcrowd and the fast flow of the rear crowd eventually make the rear crowdcatch up to the front crowd. When this situation is repeated, thedensity of the crowds increases, which results in an increase in risk.Thus, in such a situation, a rear part of the front crowd, a front partof the rear crowd, a location at which the rear crowd is expected tocatch up to the front crowd, or the like is presented as a monitoringpoint.

FIGS. 8A and 8B are diagrams illustrating the presentation of amonitoring point based on the movement of a monitoring target. In FIGS.8A and 8B, monitoring targets are crowds 50. In the case of FIG. 8A, acrowd 50-1 is moving in a rightward direction, and a crowd 50-2 inmoving in a leftward direction. In this case, the presentation unit 2060presents a frame line 30-1. The frame line 30-1 indicates a position andits vicinity where the crowd 50-1 and the crowd 50-2 are expected tointersect each other when the crowds continue proceeding.

In the case of FIG. 8B, a crowd 50-3, a crowd 50-4, and a crowd 50-5 aremoving. The presentation unit 2060 displays a frame line 30-2 at aposition and its vicinity where the crowd 50-3 and the crowd 50-4 willbe expected to intersect each other later, and displays a frame line30-3 at a position and its vicinity where the crowd 50-3 and the crowd50-5 will be expected to intersect each other later.

In this manner, when the movement of a monitoring target is handled as astate of the monitoring target, for example, monitoring pointinformation indicates information of “state: being in motion, monitoringpoint: location at which intersection is expected to occur”. As a resultof acquiring a monitoring point of “a location at which intersection isexpected to occur”, the presentation unit 2060 calculates the locationat which the intersection of the crowds is expected to occur, andperforms presentation for displaying the location.

For example, in the case of FIG. 8B, the state acquisition unit 2020acquires a state of “being in motion” with respect to each of the crowds50-3 to 50-5. The monitoring point acquisition unit 2040 acquires amonitoring point of “location at which intersection is expected tooccur” corresponding to the state of “being in motion” with respect toeach of the crowds 50-3 to 50-5. The presentation unit 2060 calculates alocation at which intersection is expected to occur, with respect toeach combination of the crowds 50-3 to 50-5, and presents a monitoringpoint in each calculated location.

Note that, the location at which intersection is expected to occur isnot limited to a location at which the crowds 50 are expected tointersect each other as in the above-described example. For example,when an obstacle such as a wall is present in the direction in which thecrowd 50 is moving, the presentation unit 2060 presents a monitoringpoint on the obstacle and its vicinity.

The movement of the crowd 50 can be calculated on the basis of a changein the position of the crowd 50 in a plurality of captured images 10 inwhich the crowd 50 is shown. Here, the movement of the crowd 50 is thespeed, moving direction, or the like of the crowd 50. Note that, amethod of calculating the movement of a crowd using a plurality ofcaptured images is well-known, and thus the detailed description thereofwill not be described.

Note that, the number of people included in the crowd 50 may be one ortwo or more.

<<Case where State of Monitoring Target is Alignment State of Queue>>

When a crowd in a captured image forms a queue, it is considered that aplace to be monitored varies depending on the state of the queue. Forexample, when the queue extends in an appropriate direction (directionalong a path or a rope, or the like), the queue is correctly aligned,and thus it may be said that there is low risk in the middle of thequeue. In this case, it is considered that the end of the queue to whicha person is newly added may be preferably monitored. Thus, themonitoring point information storage unit 3020 stores monitoring pointinformation of “state of monitoring target: queue is aligned, monitoringpoint: end of queue”.

On the other hand, for example, when the queue meanders without movingalong an appropriate direction, it is considered that a problem tends tooccur. For example, it is expected that a problem tends to occur at eachlocation at which a queue is bent. For this reason, the monitoring pointinformation storage unit 3020 stores monitoring point information of“state of monitoring target: queue is not aligned, monitoring point:location at which queue is bent and end of queue”.

Even in a state where a queue is aligned, on a platform of a station orthe like, there is narrower room around the end of a queue as the queuebecomes longer, and thus the amount of people walking the edge of theplatform increases or the amount of people attempting to pass through anarrow location around the end of the queue increases, which may resultin an increase in the risk of danger. Thus, even in a state where thequeue is aligned, the end of the queue or the edge of the platform maybe presented as a monitoring point when the length of a queue isincreased or the width of a passage around the end of the queue becomesnarrower. In this case, for example, an acceptable value of the lengthof the queue or the width of the platform may be stored in advanceinside or outside the image processing apparatus 2000.

FIGS. 9A and 9B are diagrams illustrating the presentation of amonitoring point based on the state of a queue. In FIGS. 9A and 9B, amonitoring target is a queue 60. In the case of FIG. 9A, a queue 60-1and a queue 60-2 are aligned in a rightward direction. Thus, thepresentation unit 2060 presents a frame line 30-1 and a frame line 30-2at the ends of the queue 60-1 and the queue 60-2, respectively. On theother hand, in the case of FIG. 9B, a queue 60-3 meanders. Thus, thepresentation unit 2060 presents a frame line 30-3 and a frame line 30-4at a location where the queue 60-3 meanders, and further presents aframe line 30-5 at the end of the queue 60-3.

The state of a queue can be calculated by analyzing a line of objects ina captured image 10. A method of determining “whether or not a queue isformed straight or meanders” or “whether or not the direction of a queueis an appropriate direction” is well-known. In addition, a method ofcalculating a location where a queue meanders is also well-known. Thus,these well-known methods will not be described here.

For example, the “appropriate direction of a queue” is determined inadvance in association with the camera 3040. For example, when theorientation of the camera 3040 is fixed and the position of a target forwhich a queue waits (register counter of a shop, or the like) is fixed,a direction in which the queue 60 is formed straight can be determinedin advance as an “appropriate direction of a queue”.

In addition, for example, the “appropriate direction of a queue” may becalculated using the captured image 10. Specifically, when the queue 60is formed straight, the direction of the queue 60 is calculated, and thedirection is set as an “appropriate direction of a queue”. In addition,when the queue 60 is bent in the middle thereof, for example, aguideline such as a rope is extracted from the captured image 10, and adirection along the guideline is set as an “appropriate direction of aqueue”.

<<Determination of Monitoring Point Using Plurality of Cameras>>

The image processing apparatus 2000 may use information obtained from aplurality of images captured by different cameras 3040. FIGS. 10A and10B are diagrams illustrating two captured images 10 which are capturedby two different cameras 3040. A captured image 10-1 and a capturedimage 10-2 are images obtained by capturing adjacent places, and thelower end of the captured image 10-1 is continuous with the upper end ofthe captured image 10-2.

A crowd 50-1 is shown in the captured image 10-1, and a crowd 50-2 isshown in the captured image 10-2. Here, when viewing only the capturedimage 10-1, an obstacle is not present in a movement destination of thecrowd 50-1, and thus it seems like there is no location that should bemonitored in the movement destination of the crowd 50-1. This is thesame when viewing only image 10-2. An obstacle and the like are notparticularly shown in a movement destination of the crowd 50-2 shown inthe captured image 10-2.

However, when viewing the captured image 10-1 and the captured image10-2 together, the crowd 50-1 and the crowd 50-2 can be expected toeventually intersect each other. In this manner, the image processingapparatus 2000 also performs presentation for a monitoring target thatcannot be identified unless a plurality of captured images are viewedtogether. For example, the operation is as follows.

First, the state acquisition unit 2020 acquires the state of each of themonitoring targets captured by the plurality of cameras 3040. In thecase of FIGS. 10A and 10B, the state acquisition unit 2020 acquires boththe state of the crowd 50-1 in the captured image 10-1 and the state ofthe crowd 50-2 in the captured image 10-2. Since the crowd 50-1 and thecrowd 50-2 are in motion, the monitoring point acquisition unit 2040acquires a “location at which intersection is expected to occur” as amonitoring point corresponding to each of the crowds. The presentationunit 2060 calculates a location at which the crowd 50-1 and the crowd50-2 are expected to intersect each other, from the moving directions ofthe crowd 50-1 and the crowd 50-2 and a positional relationship betweenplaces in the captured image 10-1 and the captured image 10-2. Thepresentation unit 2060 presents a frame line 30 and the like at thecalculated location. In FIGS. 10A and 10B, a frame line 30-1 ispresented in the captured image 10-1, and a frame line 30-2 is presentedin the captured image 10-2.

Note that, when there is a large number of cameras 3040, a process ofexpecting intersection which is performed for all combinations of movingmonitoring targets results in an increase in a calculation amount, andthus the time required for the process could be long. Thus, thepresentation unit 2060 may perform a process of expecting intersectionfor only monitoring targets that are located close to each other. Forexample, the presentation unit 2060 expects intersection only formonitoring targets in the respective captured images captured by theadjacent cameras 3040. In this case, state information acquired by thestate acquisition unit 2020 indicates an ID of the camera 3040 thatcaptures a monitoring target. The presentation unit 2060 obtains IDs ofthe adjacent cameras and the positional relationship therebetween usinginformation indicating the positional relationship between the cameras3040, and expects intersection. Note that, the information indicatingthe positional relationship between the cameras 3040 may be storedinside or outside of the image processing apparatus 2000.

<Details of Method of Generating State Information>

As described above, there are various states of a monitoring target.Thus, in order to generate state information, it is necessary todetermine to which state the state of a monitoring target corresponds.For that purpose, for example, a feature-value of a monitoring target(density, flow, and distribution of a crowd, and the like) which isobtained from one or a plurality of captured images is used as an input,and a model for determining whether or not the monitoring targetcorresponds to a state is generated in advance for each state. Forexample, the model is generated by being subjected to machine learningin advance using training data that includes a combination of “afeature-value of a monitoring target and a state of the monitoringtarget”. Then, a feature-value of a crowd which is obtained from acaptured image is input to a model generated for a certain state, andthus it is determined whether or not a monitoring target in the capturedimage corresponds to the state.

For example, when a feature-value of a monitoring target is input to amodel of a congestion state, an output of “being congested” or “notbeing congested” is obtained. In addition, for example, when afeature-value of a monitoring target is input to a model of a queuestate, an output of “no queue is formed”, “an aligned queue is formed”,or “a non-aligned queue is formed” is obtained.

Note that, the above-mentioned model may be configured so as tocalculate “likelihood of correspondence to a state”. For example, when afeature-value of a monitoring target is input to a model of a queuestate, the likelihood for each of the states of “no queue is formed”,“an aligned queue is formed”, and “a non-aligned queue is formed” isobtained. In this case, state information of a certain monitoring targetis generated on the basis of the likelihood obtained by inputting thefeature-value of the monitoring target to the model. Specifically, stateinformation of a certain monitoring target indicates a statecorresponding to the highest likelihood among the states correspondingto the respective likelihoods obtained by inputting the feature-value ofthe monitoring target to the model.

In addition, state information may include likelihood. In this case, thepresentation unit 2060 may present a monitoring point in considerationof the likelihood indicated in the state information. For example, thepresentation unit 2060 presents only a monitoring point corresponding tostate information in which the likelihood has a value equal to orgreater than a pre-determined value, on a presentation target image.

Note that, a crowd in a captured image may correspond to a plurality ofstates. For example, when a monitoring target forming an aligned queueis moving, the state of the monitoring target corresponds to both of “analigned queue is formed” and “being in motion”. In this case, stateinformation may be generated for all of the corresponding states, or maybe generated for only some states. When state information is generatedfor only a portion of the states, for example, the above-mentionedlikelihood is used. Specifically, there are two methods: 1) stateinformation is generated for only top n states in a case where statesare aligned in descending order of the likelihood and 2) stateinformation is generated for only states having a likelihood equal to orgreater than a pre-determined value.

Second Exemplary Embodiment

FIG. 11 is a block diagram illustrating an image processing apparatus2000 according to a second exemplary embodiment together with the useenvironment thereof. In FIG. 11, an arrow indicates a flow ofinformation. Further, in FIG. 11, each block indicates a function-basedconfiguration instead of a hardware-based configuration.

The image processing apparatus 2000 according to the second exemplaryembodiment includes a monitored position acquisition unit 2080 thatacquires a monitored position. The monitored position is a positionwhich is monitored by an observer viewing a display screen on which acaptured image is displayed. Here, a process of calculating a monitoredposition may be performed inside the image processing apparatus 2000 ormay be performed outside of the image processing apparatus 2000.

The presentation unit 2060 preferentially presents, on the presentationtarget image, a monitoring point whose presentation location is not ator around the monitored position on the display screen acquired by themonitored position acquisition unit 2080, among monitoring pointsacquired by the monitoring point acquisition unit 2040.

The monitored position is determined with various methods. Hereinafter,a method of representing a monitored position will be described.

<Determination Method Based on Observer's Eye Gaze Direction>

For example, a monitored position is determined on the basis of the eyegaze direction or the orientation of the face (hereinafter, a facedirection) of an observer. In this case, the monitored position is anintersection point between the display screen 3060 and the eye gazedirection or face direction of the observer, or the vicinity of theintersection point. Here, “the vicinity of the intersection point” is aregion having a pre-determined shape, a circular region having apre-determined radius, or the like, and the center thereof is theintersection point. In addition, a display screen may be divided intopre-determined partial regions in advance, and the partial regionintersecting the observer's eye gaze direction or the like may behandled as a monitored position. Note that, information indicating theabove-mentioned pre-determined shape and pre-determined radius may bestored in advance in a storage device provided inside or outside of theimage processing apparatus 2000.

Here, for example, when an observer monitors a certain position only fora short period of time, it is considered that the position would beinsufficiently monitored. Thus, for example, only a position that iscontinuously monitored for a pre-determined period of time or longer ora position which is monitored at a frequency higher than apre-determined frequency may be handled as a monitored position.Information indicating the predetermined period of time and thepredetermined frequency is stored in advance in a storage deviceprovided inside or outside of the image processing apparatus 2000.

For example, information that the monitored position acquisition unit2080 acquires is coordinate information indicating a monitored positionwhich is calculated by an external apparatus on the basis of the eyegaze direction, face direction of an observer, or the like. In addition,for example, information that the monitored position acquisition unit2080 acquires is information indicating the eye gaze direction or facedirection of an observer. In this case, the presentation unit 2060calculates coordinate information indicating a monitored position on thebasis of the information acquired by the monitored position acquisitionunit 2080 or the like.

For example, the eye gaze direction or face direction of an observer isrepresented by a combination of “the position of the observer, an anglein a horizontal direction, and an angle in a vertical direction”. Here,a reference of each of the angle in the horizontal direction and theangle in the vertical direction (direction handled as 0 degree) isarbitrary. In addition, a reference of the position of the observer(position handled as the origin) is arbitrary.

For example, the eye gaze direction or face direction of an observer iscalculated by capturing the face or eyes of the observer with a cameraor the like and analyzing the captured images. The camera capturing theface and eyes of the observer is installed, for example, near a displayscreen 3060. Since a technique of capturing the face or eyes of anobserver for detecting an eye gaze direction or a face direction iswell-known, the detailed description thereof will not be described here.Note that, a processing unit that detects the eye gaze direction or facedirection of an observer (hereinafter, an eye gaze direction detectionunit) may be provided inside or outside of the image processingapparatus 2000.

FIG. 12 is a diagram conceptually illustrating that a monitoring pointto be presented is determined on the basis of an observer's eye gazedirection. A captured image 10 is a captured image that the camera 3040captures and on which a frame line 30 is presented by the presentationunit 2060 after that. In FIG. 12, the captured image 10 is displayed onthe display screen 3060, and is monitored by an observer. Here, thecaptured image 10 of FIG. 12 shows the same situation as those of thecaptured image 10-1 in FIG. 9A. Specifically, two queues 60-1 and 60-2are displayed on the captured image 10.

In FIG. 12, similarly to the case of FIG. 9A, the monitoring pointacquisition unit 2040 acquires an “end” as a monitoring point for eachof queues 60. Here, an eye gaze direction 70 of an observer faces theend of the queue 60-1. In other words, the end of the queue 60-1 is theabove-described monitored position. Thus, the presentation unit 2060presents the frame line 30 at the end of the queue 60-2 which is notincluded in the monitored position or the vicinity thereof.

Note that, although the presentation unit 2060 does not present amonitoring point which is included in the monitored position or thevicinity thereof in FIG. 12, it may present a monitoring point which isincluded in a monitored position or the vicinity thereof. In this case,for example, the presentation unit 2060 emphasizes a monitoring pointnot included in a monitored position or the vicinity thereof more than amonitoring point included in a monitored position or the vicinitythereof. Here, there are various methods of emphasizing a monitoringpoint. For example, when a frame line is presented around a monitoringpoint, emphasizing is performed by increasing the thickness of the frameline. In addition, when a monitoring point is presented by changing thecolor of a monitoring point, emphasizing is performed by using a morenoticeable color. In addition, emphasizing may be performed by blinkinga frame line and region of a monitoring point. In this case, emphasizingis performed by more rapidly blinking a monitoring point as theimportance thereof becomes higher.

<Determination Method Based on Direction Indicated by Observer>

A monitored position may be determined based on a direction that anobserver's hand or finger indicates by using the similar method to themethod of determining a monitored position based on the eye gazedirection or face direction of an observer.

<Determination Method based on Position Touched by Observer>

Further, a position on the display screen 3060 which is touched by anobserver or the vicinity thereof may be handled a monitored position.The position on the display screen 3060 which is touched by the observercan be determined by, for example, using as the display screen 3060 ascreen (touch panel or the like) with which the position touched by theobserver can be detected. In addition, the position on the displayscreen 3060 which is touched by the observer can be determined by alsomonitoring the observer's movement with respect to the display screen3060 using a camera or the like. Note that, it may possible to handle asa monitored position only a position which is touched by an observer fora pre-determined period of time or longer or only a position touched ata frequency higher than a pre-determined frequency.

<Flow of Processing>

FIG. 13 is a flow chart illustrating a flow of processing performed bythe image processing apparatus 2000 according to the second exemplaryembodiment. Note that, step S102 and step S104 of FIG. 13 are the sameprocesses as step S102 and S104 of FIG. 2, respectively, and thus thoseprocesses will not be described here.

In step S202, the monitored position acquisition unit 2080 acquires amonitored position. In step S204, the presentation unit 2060preferentially presents, on the presentation target image, a monitoringpoint the location of which is not at or around the monitored positionon the display screen acquired by the monitored position acquisitionunit 2080, which monitoring point is acquired by the monitoring pointacquisition unit 2040.

<Operational Advantages>

According to the present exemplary embodiment, a monitoring point whichis not included in a position monitored by an observer or a position inthe vicinity thereof (monitored position) is displayed on the displayscreen 3060. Thus, it is achieved to prevent the observer fromneglecting monitoring by presenting a frame line or the like for amonitoring point that the observer does not monitors, and to prevent thevisibility of a monitoring location from deteriorating.

Third Exemplary Embodiment

An image processing apparatus 2000 according to a third exemplaryembodiment has the same configuration as that of the image processingapparatus 2000 according to the first or second exemplary embodiment,and has the same function as that of the image processing apparatus 2000according to the first or second exemplary embodiment except for thefollowing description.

In the third exemplary embodiment, a monitoring point is associated witha state of a monitoring target and another element (hereinafter, asecond element). The monitoring point acquisition unit 2040 acquires thestate of the monitoring target and the second element, and acquires amonitoring point corresponding thereto. Hereinafter, a specific exampleof the above-mentioned second element will be described.

<Schedule Information>

For example, a second element is schedule information indicating a date,a day of the week, a time, or the like. There are locations for whichthe necessity of monitoring varies according to the situations; somelocations are safe in the daytime but are dangerous in the nighttime,other locations are safe on weekdays but are dangerous on weekends andon national holidays.

Thus, the monitoring point information storage unit 3020 stores amonitoring point in association with the state of the monitoring targetand schedule information. The monitoring point acquisition unit 2040acquires schedule information in addition to the state of the monitoringtarget, and acquires a monitoring point corresponding to a combinationthereof. Thereby, it is possible to acquire an appropriate monitoringpoint based on time.

<Weather Information>

For example, a second element is weather information indicating weather(the weather, temperature, humidity, wind velocity, or the like). Forexample, it may be necessary to be careful for a location in whichpuddles are easily generated there in rainy weather while the locationis safe in fine whether.

Thus, the monitoring point information storage unit 3020 stores amonitoring point in association with the state of a monitoring targetand weather information. The monitoring point acquisition unit 2040acquires weather information in addition to the state of the monitoringtarget, and acquires a monitoring point corresponding to a combinationof the state and the information. Thereby, it is possible to acquire anappropriate monitoring point based on the weather at the time ofmonitoring. Note that, weather information to be acquired may beinformation indicating the actual weather at that time or may beprediction information of weather indicated by a weather forecast or thelike.

<Event Information>

In addition, for example, a second element is information regarding anevent that is performed at or around a monitoring place or the like(hereinafter, event information). In a case of monitoring a place inwhich any event is held, such as a stadium in which a game is played, aplace to be monitored may varies according to the state of the event andthe result thereof. For example, in the case of a stadium in which agame is played, there is high possibility of that a large number ofirritated supporters exist at rooters' seats of a team that has lost thegame, around a passage toward an exit from the rooters' seats, or thelike, which results in a high necessity of monitoring.

Thus, the monitoring point information storage unit 3020 stores amonitoring point in association with the state of a monitoring targetand event information. The monitoring point acquisition unit 2040acquires event information (information regarding which team has won,and the like) at the time of monitoring in addition to the state of amonitoring target, and acquires a monitoring point corresponding to acombination of the state and the information. Thereby, it is possible toacquire an appropriate monitoring point based on conditions of an eventat the time of monitoring. Note that, event information to be acquiredmay be information indicating actual conditions of an event at thattime, or may be expected conditions of an event. For example, when anevent is a game and monitoring is performed at the time of a final stageof the game, the “expected conditions of an event” are a result of thegame which is expected on the basis of the situation of the game at thattime, and the like.

<Flow of Processing>

FIG. 14 is a flow chart illustrating a flow of processing performed bythe image processing apparatus 2000 according to the third exemplaryembodiment. Note that, step S102 and S106 of FIG. 14 are the sameprocesses as step S102 and S106 of FIG. 2, respectively, and thus thoseprocesses will not be described here.

In step S302, the monitoring point acquisition unit 2040 acquires asecond element. In step S304, the monitoring point acquisition unit 2040acquires a monitoring point corresponding to the state of a monitoringtarget and the second element.

<Operational Advantages>

According to the present exemplary embodiment, a monitoring point isacquired in consideration of a second element such as weatherinformation, in addition to the state of a monitoring target. In thismanner, it is possible to present an indication of a monitoring pointwhich is more appropriate for conditions at that time, as compared to acase where a monitoring point is acquired solely on the basis of thestate of a monitoring target. As a result, an observer can perform moreappropriate monitoring in accordance with conditions at the time ofmonitoring.

Fourth Exemplary Embodiment

FIG. 15 is a block diagram illustrating an image processing apparatus2000 according to a fourth exemplary embodiment. In FIG. 15, an arrowindicates a flow of information. Further, in FIG. 15, each blockindicates a function-based configuration instead of a hardware-basedconfiguration.

The image processing apparatus 2000 according to the fourth exemplaryembodiment includes a state acquisition unit 2020 described in the firstexemplary embodiment and a monitored position acquisition unit 2080described in the second exemplary embodiment. Further, the imageprocessing apparatus 2000 according to the fourth exemplary embodimentincludes a monitoring point information generation unit 2100. Themonitoring point information generation unit 2100 handles a monitoredposition acquired by the monitored position acquisition unit 2080 as amonitoring point, and generates monitoring point information byassociating the state of a monitoring target acquired by the stateacquisition unit 2020 with the above-mentioned monitoring point. Themonitoring point information generated by the monitoring pointinformation generation unit 2100 is stored in a monitoring pointinformation storage unit 3020.

Suppose that information indicating the state of a monitoring target(state information) indicates a state of “a queue is aligned”, and thatan observer is monitoring the end of a queue. In this case, themonitoring point information generation unit 2100 generates monitoringpoint information by associating a monitoring point of the “end” withthe state of “a queue is aligned”.

<Association Between State Information and Monitored Position>

For example, the monitoring point information generation unit 2100associates pieces of state information with monitored positions in orderfor the acquisition thereof. Specifically, the monitoring pointinformation generation unit 2100 generates monitoring point informationby associating n-th state information with an n-th monitored position.This method is used, for example, when pieces of state information andmonitored positions are sequentially acquired in real time.

In addition, for example, the monitoring point information generationunit 2100 may associate state information with a monitored positiongenerated at the time close to the time when the state information isgenerated. In this case, the state acquisition unit 2020 acquires stateinformation in association with the time when the state information isgenerated. Similarly, the monitored position acquisition unit 2080acquires a monitored position in association with the time when theposition is generated. Note that, the phrase “the time of generationbeing close to each other” means that, for example, a difference betweenthe times of generation is equal to or less than a pre-determined time.For example, this method is used when a plurality of pieces of stateinformation and monitored positions are temporarily stored in a storagedevice or the like and then monitoring point information is generatedthrough batch processing. However, this method may be used when piecesof state information and monitored positions are sequentially acquiredin real time.

<Use of Monitoring Results of Plurality of Monitoring People>

The monitoring point information generation unit 2100 may generatemonitoring point information on the basis a plurality of monitoringpeople's monitoring actions. In this case, the monitoring pointinformation generation unit 2100 may determine the importance of amonitoring point and may generate monitoring point information solelywith respect to monitoring points having high importance. Specifically,first, the monitoring point information generation unit 2100 generates acandidate of monitoring point information (hereinafter, candidateinformation) by associating “the state of a monitoring target acquiredby the state acquisition unit 2020, a monitored position acquired by themonitored position acquisition unit 2080, and an ID of an observerhaving performed monitoring” with each other. When the number of piecesof candidate information indicating the same combination of “the stateof a monitoring target and a monitored position” is equal to or greaterthan a pre-determined number, the monitoring point informationgeneration unit 2100 handles the combination as monitoring pointinformation. In this case, the number of pieces of candidateinformation, that is, the number of monitoring people who monitor acertain monitored position is handled as importance of a monitoringpoint corresponding to the monitored position.

In the above-mentioned method, weighting may be performed in associationwith an ID of an observer. Specifically, the monitoring pointinformation generation unit 2100 sums the weight of an observercorresponding to each piece of candidate information each of which showsthe same combination of “the state of a monitoring target and amonitored position”. When the sum is equal to or greater than apre-determined value, the combination of “the state of a monitoringtarget and a monitored position” is handled as monitoring pointinformation. For example, the weight corresponding to an ID of anobserver is determined in advance in accordance with the level of theobserver's skill. In this case, the sum calculated for a certainmonitored position is handled as importance of a monitoring pointcorresponding to the monitored position.

<Flow of Processing>

FIG. 16 is a flow chart illustrating a flow of processing performed bythe image processing apparatus 2000 according to the fourth exemplaryembodiment. In step S402, the state acquisition unit 2020 acquires thestate of a monitoring target in a captured image. In step S404, themonitored position acquisition unit 2080 acquires a monitored position.In step S406, the monitoring point information generation unit 2100generates monitoring point information by associating a monitoredposition acquired by the monitored position acquisition unit 2080 withthe state of a monitoring target acquired by the state acquisition unit2020.

<Operational Advantages>

According to the present exemplary embodiment, monitoring pointinformation is generated on the basis of the state of a monitoringtarget in a captured image and a position monitored by an observer whois monitoring the captured image. In other words, the observer'smonitoring action is learned by the image processing apparatus 2000, andmonitoring point information is generated as a result of the learning.In this manner, labor required for manually creating monitoring pointinformation is reduced.

For example, as a method of operating the image processing apparatus2000 of the present exemplary embodiment, an operation method ofgenerating monitoring point information by learning a highly skilledobserver's monitoring action is considered. The monitoring pointinformation generated in this manner is used by the image processingapparatuses 2000 according to the first to third exemplary embodiments.In this manner, monitoring people can share information regarding “whatlocation a highly skilled observer is monitoring in what state of amonitoring target”. Accordingly, even when the observers' skills vary,it is possible to efficiently perform monitoring without neglecting. Inaddition, it is possible to reduce training costs required to train anew observer.

The exemplary embodiments of the invention have been described so farwith reference to the accompanying drawings. However, the exemplaryembodiments are merely illustrative of the invention, and other variousconfigurations can also be adopted.

Hereinafter, examples of reference configurations will be added.

(1) An image processing apparatus including:

a state acquisition unit acquiring a state of a monitoring target in acaptured image captured by a camera;

a monitoring point acquisition unit acquiring a monitoring pointcorresponding to the state of the monitoring target acquired by thestate acquisition unit, the monitoring point indicating a position to bemonitored in the captured image, and being acquired from a monitoringpoint information storage unit storing the state of the monitoringtarget and the monitoring point in association with each other; and

a presentation unit presenting the monitoring point acquired by themonitoring point acquisition unit on a first captured image captured bythe camera.

(2) The image processing apparatus according to (1), further including amonitored position acquisition unit acquiring a monitored position,which is a position monitored by an observer viewing a display screen onwhich the captured image is displayed,

wherein the presentation unit preferentially presents, on the firstcaptured image, a monitoring point whose presentation location is not ator around the monitored position on the display screen, the monitoringpoint being acquired by the monitoring point acquisition unit.

(3) The image processing apparatus according to (2), wherein themonitored position acquisition unit handles, as the monitored position,any one or more of: a position on the display screen corresponding to aneye gaze direction of the observer; a position on the display screencorresponding to a direction indicated by a finger or hand of theobserver; and a position on the display screen touched by the observer.

(4) The image processing apparatus according to any one of (1) to (3),wherein the state acquisition unit acquires a degree of congestion of acrowd as the state of the monitoring target.

(5) The image processing apparatus according to any one of (1) to (4),wherein the state acquisition unit acquires a movement state of a crowdas the state of the monitoring target.

(6) The image processing apparatus according to any one of (1) to (5),wherein the state acquisition unit acquires an alignment state of aqueue as the state of the monitoring target.

(7) The image processing apparatus according to any one of (1) to (6),

wherein the monitoring point information storage unit further stores thestate of the monitoring target and the monitoring point in associationwith schedule information, and

wherein the monitoring point acquisition unit acquires the monitoringpoint corresponding to the state of the monitoring target acquired bythe state acquisition unit and schedule information corresponding to atime at which the captured image is captured.

(8) The image processing apparatus according to any one of (1) to (7),

wherein the monitoring point information storage unit further stores thestate of the monitoring target and the monitoring point in associationwith weather information, and

wherein the monitoring point acquisition unit acquires the monitoringpoint corresponding to the state of the monitoring target acquired bythe state acquisition unit and weather information indicating weather ata time of the captured image being captured.

(9) The image processing apparatus according to any one of (1) to (8),

wherein the monitoring point information storage unit further stores thestate of the monitoring target and the monitoring point in associationwith a state of an event occurring at a place where the camera isinstalled, and

wherein the monitoring point acquisition unit acquires the monitoringpoint corresponding to the state of the monitoring target acquired bythe state acquisition unit and the state of the event at the time of thecaptured image being captured.

(10) An image processing apparatus including:

a state acquisition unit acquiring a state of a monitoring target in acaptured image captured by a camera;

a monitored position acquisition unit acquiring a monitored position,which is a position monitored by an observer viewing a display screen onwhich the captured image is displayed; and

a monitoring point information generation unit handling the monitoredposition as a monitoring point, and generating monitoring pointinformation by associating the monitoring point with the state of themonitoring target, the monitoring point indicating a position to bemonitored in the captured image.

(11) The image processing apparatus according to (10),

wherein the monitored position acquisition unit acquires monitoredpositions with respect to a plurality of observers, and

wherein the monitoring point information generation unit determinesimportance of a monitoring point corresponding to the monitored positionbased on the number of the observers having monitored the monitoredposition.

(12) The image processing apparatus according to (10),

wherein the monitored position acquired with respect to an observer isweighted based on a level of a skill of the observer, and

wherein the monitoring point information generation unit determinesimportance of a monitoring point corresponding to the monitored positionbased on a weighted sum of the monitored position.

(13) A monitoring system including the camera, the monitoring pointinformation storage unit, and the image processing apparatus accordingto any one of (1) to (9).

(14) A monitoring system including the camera and the image processingapparatus according to any one of (10) to (12).

(15) An image processing method performed by a computer, the methodincluding:

acquiring a state of a monitoring target in a captured image captured bya camera;

acquiring a monitoring point corresponding to the state of themonitoring target acquired by the step of acquiring the state of themonitoring target, the monitoring point indicating a position to bemonitored in the captured image, and being acquired from a monitoringpoint information storage unit storing the state of the monitoringtarget and the monitoring point in association with each other; and

presenting the monitoring point acquired by the step of acquiring themonitoring point on a first captured image captured by the camera.

(16) The image processing method according to (15), further includingacquiring a monitored position, which is a position monitored by anobserver viewing a display screen on which the captured image isdisplayed,

wherein the step of presenting the monitoring point preferentiallypresents, on the first captured image, a monitoring point thepresentation location of which is not at or around the monitoredposition on the display screen, the monitoring point being acquired bythe step of acquiring the monitoring point.

(17) The image processing method according to (16), wherein the step ofacquiring the monitored position includes handling, as the monitoredposition, any one or more of: a position on the display screencorresponding to an eye gaze direction of the observer; a position onthe display screen corresponding to a direction indicated by a finger orhand of the observer; and a position on the display screen touched bythe observer.

(18) The image processing method according to any one of (15) to (17),wherein the step of acquiring the state of the monitoring targetincludes acquiring a degree of congestion of a crowd as the state of themonitoring target.

(19) The image processing method according to any one of (15) to (18),wherein the step of acquiring the state of the monitoring targetincludes acquiring a movement state of a crowd as the state of themonitoring target.

(20) The image processing method according to any one of (15) to (19),wherein the step of acquiring the state of the monitoring targetincludes acquiring an alignment state of a queue as the state of themonitoring target.

(21) The image processing method according to any one of (15) to (20),

wherein the monitoring point information storage unit further stores thestate of the monitoring target and the monitoring point in associationwith schedule information, and

wherein the step of acquiring the monitoring point includes acquiringthe monitoring point corresponding to the state of the monitoring targetwhich is acquired in the step of acquiring the state of the monitoringtarget and schedule information corresponding to time at which thecaptured image is captured.

(22) The image processing method according to any one of (15) to (21),

wherein the monitoring point information storage unit further stores thestate of the monitoring target and the monitoring point in associationwith weather information, and

wherein the step of acquiring the monitoring point includes acquiringthe monitoring point corresponding to the state of the monitoring targetwhich is acquired in the step of acquiring the state of the monitoringtarget and weather information indicating weather at a time of thecaptured image being captured.

(23) The image processing method according to any one of (15) to (22),

wherein the monitoring point information storage unit further stores thestate of the monitoring target and the monitoring point in associationwith a state of an event that occurs at a place where the camera isinstalled, and

wherein the step of acquiring the monitoring point includes acquiringthe monitoring point corresponding to the state of the monitoring targetwhich is acquired in the step of acquiring the state of the monitoringtarget and the state of the event at a time of the captured image beingcaptured.

(24) An image processing method performed by a computer, the methodincluding:

acquiring a state of a monitoring target in an image captured by acamera;

acquiring a monitored position, which is a position monitored by anobserver viewing a display screen on which the captured image isdisplayed; and

handling the monitored position as a monitoring point, and generatingmonitoring point information by associating the monitoring point withthe state of the monitoring target, the monitoring point indicating aposition to be monitored in the captured image.

(25) The image processing method according to (24),

wherein the step of acquiring the monitored position includes acquiringa monitored position with respect to a plurality of observers, and

wherein the step of generating monitoring point information includesdetermining importance of a monitoring point corresponding to themonitored position based on the number of the observers having monitoredthe monitored position.

(26) The image processing method according to (25),

wherein the monitored position acquired with respect to an observer isweighted based on a level of a skill of the observer, and

wherein the step of generating monitoring point information includesdetermining importance of a monitoring point corresponding to themonitored position based on a weighted sum of the monitored position.

(27) A program causing a computer to operate as the image processingapparatus according to any one of (1) to (12).

It is apparent that the present invention is not limited to the aboveembodiment, and may be modified and changed without departing from thescope and spirit of the invention.

This application claims priority from Japanese Patent Application No.2014-152336, filed on Jul. 25, 2014, the entire contents of which areincorporated herein.

The invention claimed is:
 1. A monitoring system including: at least onenon-transitory computer-readable medium storing instructions; and atleast one processor configured to execute the instructions to perform:acquiring a degree of congestion of a crowd in an image which is one ofcaptured images, wherein the degree of congestion of the crowd is one ofthree or more levels; acquiring a monitoring point that varies dependingon the degree of congestion of the crowd, the monitoring pointindicating a position to be monitored in the image, wherein themonitoring point corresponds to one of the three or more levels ofdegree of congestion of the crowd; and presenting the monitoring pointon a first captured image which is one of captured images.
 2. Themonitoring system according to claim 1, wherein when the degree ofcongestion of the crowd is lower than a threshold, acquiring an entranceas the monitoring point corresponding to the degree of congestion of thecrowd, and when the degree of congestion of the crowd is not lower thanthe threshold, acquiring, as the monitoring point, an area where peopleare crowded or an area where people pass.
 3. The monitoring systemaccording to claim 1, wherein the correspondence between the degree ofcongestion of the crowd and the monitoring point is determined on thebasis of characteristics of a place in the captured image, orcharacteristics of people using the place.
 4. The monitoring systemaccording to claim 1, wherein the at least one processor is furtherconfigured to perform: acquiring a monitored position, which is aposition monitored by an observer viewing a display screen on which thecaptured image is displayed; and preferentially presenting, on the firstcaptured image, a monitoring point whose presentation location is not ator around the monitored position on the display screen.
 5. Themonitoring system according to claim 4, wherein the monitored positionincludes any one or more of: a position on the display screencorresponding to an eye gaze direction of the observer; a position onthe display screen corresponding to a direction indicated by a finger orhand of the observer; and a position on the display screen touched bythe observer.
 6. The monitoring system according to claim 1, furthercomprising: a monitoring point information storage storing the degree ofcongestion of the crowd and the monitoring point in association witheach other.
 7. The monitoring system according to claim 6, wherein themonitoring point information storage further stores the degree ofcongestion of the crowd and the monitoring point in association withschedule information, and the at least one processor is configured toperform acquiring the monitoring point corresponding to the degree ofcongestion of the crowd and the schedule information corresponding to atime at which the captured image is captured.
 8. The monitoring systemaccording to claim 6, wherein the monitoring point information storagefurther stores the degree of congestion of the crowd and the monitoringpoint in association with weather information, and the at least oneprocessor is configured to perform acquiring the monitoring pointcorresponding to the degree of congestion of the crowd and the weatherinformation indicating weather at a time of the captured image beingcaptured.
 9. The monitoring system according to claim 6, wherein themonitoring point information storage further stores the degree ofcongestion of the crowd and the monitoring point in association with astate of an event occurring at a place where a camera is installed, andthe at least one processor is configured to perform acquiring themonitoring point corresponding to the degree of congestion of the crowdand the state of the event at the time of the captured image beingcaptured.
 10. A monitoring method including: acquiring a degree ofcongestion of a crowd in an image which is one of captured images,wherein the degree of congestion of the crowd is one of three or morelevels; acquiring a monitoring point that varies depending on the degreeof congestion of the crowd, the monitoring point indicating a positionto be monitored in the image, wherein the monitoring point correspondsto one of the three or more levels of degree of congestion of the crowd;and presenting the monitoring point on a first captured image which isone of captured images.
 11. The monitoring method according to claim 10,wherein when the degree of congestion of the crowd is lower than athreshold, acquiring an entrance as the monitoring point correspondingto the degree of congestion of the crowd, and when the degree ofcongestion of the crowd is not lower than the threshold, acquiring, asthe monitoring point, an area where people are crowded or an area wherepeople pass.
 12. The monitoring method according to claim 10, whereinthe correspondence between the degree of congestion of the crowd and themonitoring point is determined on the basis of characteristics of aplace in the captured image, or characteristics of people using theplace.
 13. A non-transitory computer-readable medium storing programscausing a processing device to perform: acquiring a degree of congestionof a crowd in an image which is one of captured images, wherein thedegree of congestion of the crowd is one of three or more levels;acquiring a monitoring point that varies depending on the degree ofcongestion of the crowd, the monitoring point indicating a position tobe monitored in the image, wherein the monitoring point corresponds toone of the three or more levels of degree of congestion of the crowd;and presenting the monitoring point on a first captured image which isone of captured images.
 14. The non-transitory computer-readable mediumaccording to claim 13, wherein the program causes the processor deviceto perform: when the degree of congestion of the crowd is lower than athreshold, acquiring an entrance as the monitoring point correspondingto the degree of congestion of the crowd, and when the degree ofcongestion of the crowd is not lower than the threshold, acquiring, asthe monitoring point, an area where people are crowded or an area wherepeople pass.
 15. The non-transitory computer-readable medium accordingto claim 13, wherein the correspondence between the degree of congestionof the crowd and the monitoring point is determined on the basis ofcharacteristics of a place in the captured image, or characteristics ofpeople using the place.